Embedding as a modeliilg problem

Standard appronchcs to time-dclay cmbcdding will oflc11 fa i l to provide an elnberidi~ig that is uschil for nii\ny commoa applicaliot~s. 'This Ilappws in particular whcr~ t lwc ar t multiple timescalcs in tile clyr~:imics. \Vc prcsent a ~nodified procedure, P I O I I I I I I I J U ~ R L e~rlbedditlfi, which ovcrcornes si~ch problcms in many cascs For rnorc carnplcx ~mdincar clynalnics we ~niroduce vclrirlhle e m b c d d i ~ ~ ~ , where, in a ~uitablc sensc. ~hee~nbctlriing changes with [he statc of lhesystem. Wc sllow how to ilnplcmcnt Ihese procedures by combining ctnbeiklirlg and nodel ling into a si~iglc procctlure with a single nptitni-/.:~liun goal. O 1998 Elsevier Scicncc B.V. I . EmbctIding and nlodelirlg ?'his paper concerns lhc nr~nlysrs of limc sclies, in pa~~iculnr, those that itre recordings fro111 rwnt ineat dynarnical systems. In reccnt years, many new timescrics a~ialys~s and modeling techniques have bcen derived from nonlinear dyrlarnical syslclns thcory. A common procedural fcature of these tcchniques is ernlrcddit~g, which IS performed beforc, A~IJ is iluitc clistir~ct from, the modcl~ng process. The pracI (ice of embedding rcsts up011 a celebrated thcoretn of Tnkcns [201 ant1 a ru~mbcr of srgriilicnnt algorithms related to it [2,4,7]. It 1s well known [hat therc are <lifficulties with thc c~nbedding stcp that precedes ~nodcl~lug bur until recently no hc~ter nltc~natrve had suggested itself. Here wc prcsent an alternative that appeals lo be powerfill an~l widely applicable. Instead of treating embedding and modelii~g ns ~ w i l distinct proccdurcs, wc cornbinc cmbcdding 2nd modcling into a siligle procedure with a single optirnization criterion in r i sensc. Ihc tequi~.enlerlt of a cnrefiiIly constructed global embcdding of thc dy namical systcm is dispensed with, since for lIlc purposes of tinlc series prcdiclio~~ sucl~ an embedding ir not needed, as wc shall sec. Our alternative przlcediires build cilher nun-uniform global ernbcdding~, or non-uniform locaI cmbeddirlgs t t ~ t vary with systzm state, lhe embcdd~ngs in either case being chosen s o as to optimizc n modeling criterion. In order to uuderstanrl thc dcvclopmenl of our rnorleling proccdurcs, one has lo be familiar w ~ t h the minimum dcscriplion lengt 11 (M DL) principle for i~lodeling time ~erics. The main part of this paper begins (Scctiorl 2) wit11 a brief description of [he MDL principle and iln algorithm for itnplcli~cnlirig i t . Secrion 3 is a critiqi~e of commoi~ly acccp~ed cmI~ccldr ng procedurct from a motleling pilint of vicw, 0 167-27801981% 19.00 O 1998 El.rcvizr Scicncc B.V. Al l rights rcscrvctl. PI[: SO 167-2739(98)000#9-X i \ l parliciilar EI-om Ihe standpoint of liune sc~.ies prcdicrioa. Scction 4 describes how lo avoid sowc of thc cliffiliculries we identify+ iisiug ~aon-wa(fo~,~n cmhcd(lings, As ii spccial casc we descr'ihe [he bi~ilding of lincar ~notlels or I I I I I ~ series; this results ill il I ~ I I C ~ T luodeling tcchtliquc, rr.rtrtcec1 rru~oregte.e.rsive modcIi~ig, wllicl~ has ulvantagcs over sc;i~~dnl.d linei~r nutorcgrtssive modcllng. The ideas on linear tllodcling arc 1101 ccntr;~l to ltic tliesis wc are tlcvcloping; (hey arc a special c : ~ that merits dcscriprion i n its own right. Scc tion 4 iiiclutlcs applicatio~~s of nu~~-uuiform cinbedding to both lincar and nanlincar motlcling of the suasllot time series, a~ id to rccorciiogs of inFi[;lnt breathiug diiritlg natural slccp. Scr.rion 5 dcscribcs modcls using vcrricrbit t'/r~Clerlili~lg.s, which can bc tl~ougbt or as u sct u l lucal embcd(lings that vary with thc systcm statc. Whet1 vi~rirtblc cmhcdding is appltod to rmadial-basis modcls, which arme il piirtiulllilt nonlinear* i~~odelirlg tccb~liquc, or~c arrivcs at ii natural gerlcr,alizrttiun we call cyiit~dr.icol b0si.r {nodcis. We dcscribc ijn application to the ]nod+ cling of a spoken vowel; in e f f tc~, wc arc producirig n pl~e~~ornc~~ological ~ncldcl of tlie r l y ~ ~ i ~ ~ n i c s 01' thc vocnI irxcl as uscd ia protiucing that, pnrtic~~lnr sourtri. We also considclihc i~np~.ovcd ability of no~i l ine~r ~nodcls wit11 viiriabIe elnbtddings to caplure Lhe c-l y~irlrnics of a systcln from a lime sciics by comparing r,xrlial and cylindrical basis modcls of thc su~~vpot ti~rle scrics. Thc p~'incipal motivation of this work is 1 1 1 ~ ciesire to obtain 11~odc1.s tlut accurntcly reflect thc tlyniunics of thc sysrcm being ~no;Ielcd. That is, a n~odcl sho~~l t l 1101 onIy l i t data 3rd predict it wcll, but should itlso I~avc d y r ~ a ~ ~ ~ i c a l bch vior Iikc that of thc measu~~etl system. This is n stringent critcrioa ihnt is satisfictf by vcry fcw modclitlg methods; 111c work describcrl t~crc pl,ovirlcs oaly n partial soliitioo. 2. Thc rni~ii~nuln description Ic~~gth principle Thc work dcscribcti in diis pihpc~, rclics on two thew ~eficrrl :n\d procedural tools: the tnir~irtium ricrcriptiori k t rgrh principlc, ant1 algoritllrn lor building rnodels it1 a gctleral class of ~~aulinear n10dc1.s called pszrrtlolifzear modcls IVe havc cicscribed [hcse in otlicr publicntioris ;t111( t.c~ifc~.s are rcfer.~-cd to tlwm for greater dctnil chan is prnvided here [ 10,l I], The minirnum dcscril~timn Icngth principlc is all application of Occam's ru1or i n modeling cootcat: i l dcti~~c:: the bcst niotlel [or. ;i titilc scrim to bc the uric [hat i~cliicvcs t l~c most concise dcscriptioa or thc di~ta. 'To understand how lhz principle works, suppose you (ihc "sc~i(lcl") t~avc collected an cxperi~~icntnl titoc scries x ( / ) , r = I , . , . , n , mensurcd to ijn accuracy of (say) L2 bits, ;lnd you wish to commm~icatc tliis clnl;) to A collenguc ( the "recipicnt"). You could send the raw data. Altcmativcly, you couid construct a dyniirnical model from tta data ch;~ eilitblc.; thc rccipieut to gl'crlict a vnluc of x ( r ) From carlicr val~ms. [t' you n ~ ~ d your collcugue have previously agrccd 011 ;1 c1i1s.s of rnorleIs, then you co~~l t l colmiiunicatc the dii~ii by srnding the pnmmctc1.s of a model, c ~ ~ o u g h illitin1 da[a to start p~ulicting f11~u1.r: volucs of the titnc serics, and the errors bctwcen thc truc time sc~icr autl Ihc values pl-edictcd by the tnotlel. Given this rnformation, thc recipicnt ~ 3 1 1 rcconstrllct ~ I I C expcritncnlnl d:lta to its full ~ncnsurcrl accuracy. An important point is t h n ~ dir: pammctcrs anrl crrors l~ecd only be specified to lini~c accuincy. Pui~thcrmorc, ii Ihc inotiel is good, then [ l~c tvrnl number of bits rcquil-cd to triuumit paramcrcrs, initi:il values anil errors will bc lcss than thc nu~nbcr 01 hits of riiw dnta. In practice (tic miaimurn dcscr.iptiun Icngrh principle I-cquircs calcul~~ting an approximatlo11 LO Ille descril)liot~ I r * r i ~ r l t ol D I ~ timc series ant1 model, which is effectively thc number of hits rcquir.erl to transmit the motlel plus the ~ lu~nber uf bits rcrluircd to Lransmit thc errors. (Thc initial conrlirious arc included in thc par;lructzr count, Ihough tlicir effcct orily mnttc~ s when we iuc comparing ciiffcrcnt cmbedtling di~ncnsions ) Undci. Fairly gcnct-a1 assumptions o ~ i c citll write (Dcscriptioo length) * (number O F data) --.. ' Rissnlle~~ 1161 plcsc~lrs arjotljcr way tu lint1 nlu~lcls rvilh l l~ i r l i r l~a l dcscriplinlr Ic11g111, using so-cnllotl "honcsl" prcdiclrou crrors. This i ~ r 11 utluccs t\lflicul~ics ill IILI lltli~jg ij01lli11cilr 111odcI<. aucl ivc Icauc it Tor fulurc work. x lo_c(Mcan s(llIi1rC prediction error) -b (Pa~iilty fur nutrtbel. and ncctuncy of paramc~crs) . As t t ~ c r~umbcr of paramele1.s ill a rltudcl irlcrcascs the prediction errors dccrcnse, but evcnru;~lly. ~ h c pzt~r~lty for introtluciag ~inolhcr parameter ou[weiglw thc hcncfi t it has ia retluclng p~.cdictiou clyurs. Thc otoJcl that altains the minimuill description Icngth is dcf a r J to be the optimal inodcl within lhc class of models considered. We do not have spacc hcrc to tliscuss it1 tictail why Ihis is succcssfi~l; ex~ensive disci~ssions arc to bc Fourld clscwherc [ l0,IG.Z l I. 111 spccial irtodel classes, zxplicit approxitnaliolls to the description Icng[l~ call bc. cnlculated A part~mlarly usefu I class of paramztri./cd nonlinear autu~.egr.essive ~notlel consisrs o l lllose wc caIl p.~el~lio-litrer~r. ~nodeIs, which hnvc (he forrn tot. some sclcction of nonIi~leiw fu~ lc t~u~ i s fi, unknow~l paramelers Ai and unknown i.i d. rar~dom varliltcs t , . (Observe In passing tllntctloosil~g V ( I ) ntnounts ta using a particular embedding.) Dcfinc V, =l ,h (v ( I ) ) , . . . , h(v(rl))jr, i = l , . . . , In, (3) y = (~ (11 , . . . , ~ ( F l j ) ' ~ , (4) 'I' A = ( k l , . . . , A I , , ) (3 and let V hc thc mu:rix wllose coluo~ns are Vi, i = I , . . , , t l a . If tllc F, arc ussumctl to be Gaussian and h llas ~ C C I I clloscn to tnillirnizc tlh: son1 of squares of the prcdiction elrors e = y VX, rhzn thc dcscription length 1101 is houndeci by wl~crc k is tbc number of nou-zero cornponcnts of A , y is rclntcd to the scalc of the data aud 6 solves [QS I j = I/$, wherc nail v is cnmposed of just thosc ca lu~l~ns of V lhat corrcspotld to nov-zero elelnenls of h. 'The variables 6 can bc intcrprctcd as thc relative precisions to which the pammekrs h are speciticd. Thc altr.nclion of pscudo-liocar mudcls is thnl the pal.;lmeters h arc easily ci~lculi\~?d, sinzc Ihc surxl of squares of thc prediclion errors r = y V A Cilrl bc ~ninilnizctl cflicicntly using siugular value decomposition or any of its many cquivalei~ts. What makes general pseurlo-linear rnodels different from, and 1noi.l: powcrfiil rtla11, special cilscs such as l i~war or global poly~lornial models, is that the bmis fiirlctions fi can bc chosen in many ways. TI12 cr-itical prohlem i s therefore Ilow to sclcct thc b;isis flncrions I;, which will, in gcncral, be nonlinen1 ft~actir>ns depcntiing on various parameters over which thcy arc optirni~ed. UnSortiulatcly, (his opli~nization is noniinear and so is in gc~lcral d~ff~cult , slow and prone to captilrc by local ~ n i n ~ t t ~ a . (This problcm is well-known in modcliog v ia single-layer neural ncts, a particular pseudo-linc:~r nl~proach.) Instead of optitnixing the paromc~crs of a few basis functions, we caa iasteacl gcncralc a Iiirge number of fixed basis functions, not only at the stnrl but also atlal~tively as llw modelbuilding progresscs, and scl e c ~ n subset of lhcm that optimizes the dcscription Icngth. 'Chis i~lternativc schzme requircs an efficient cornhinaturinl optimiziition tnethod to sclcct an optirunl subset of thc hasis futtct~ons. 11 woulri BPPC:II that we hnvc made the probictn worse, since combinatorial optimizatic~n i s aotot.iously hard, hut i11 fact thc following subset selection algoritlirn, dcscrihcd in dcbail elscwhcr~ [ t o ] is very succcssftil in ~nnst of thr: applications wc have considcrcd. Thc algoritlttn rclects subsets that are near-oplin~nl accu~ding to t t x (lcscriptiotl Length criterion, and hcnw prnduocs good pscudo-lincat models. It opcrntcs by adding 2nd rzri~oving candidate functions froll~ u given basis scr il~cording ro a local opti tnality criterion, and acccpling n set of given sizc as opti~nnl i f thc same ciindidate is rclnovc(l as was just addcd. Thc sizc of thc basrs set is inc~,cascd until the tlcscription length criterion says i t hrn becotnc too Inrgc, and then tlw besl set found so far is s ~ l ~ ~ t c r l ;IS t R t Ovt r i l l l opti3, R~nbcdrli~~g